Processing Data-Intensive Workflows in the Cloud

In the recent years, large-scale data analysis has become critical to the success of modern enterprise. Meanwhile, with the emergence of cloud computing, companies are attracted to move their data analytics tasks to the cloud due to its exible, on demand resources usage and pay-as-you-go pricing model. MapReduce has been widely recognized as an important tool for performing large-scale data analysis in the cloud. It provides a simple and fault-tolerance framework for users to process data-intensive analytics tasks in parallel across dierent physical machines. In this report, we survey alternative implementations of MapReduce, contrasting batched-oriented and pipelined execution models and study how these models impact response times, completion time and robustness. Next, we present three optimization strategies for MapReduce-style workows, including (1) scan sharing across MapReduce programs, (2) workow optimizations aimed at reducing intermediate data, and (3) scheduling policies that map work ow tasks to dierent machines in order to minimize completion times and monetary costs. We conclude with a brief comparison across these optimization strategies, and discuss their pros/cons as well as performance implications of using more than one optimization strategy at a time.University of Pennsylvania Department of Computer and Information Science Technical Report No. MS-CIS-12-07. Comments University of Pennsylvania Department of Computer and Information Science Technical Report No. MSCIS-12-08. This technical report is available at ScholarlyCommons: http://repository.upenn.edu/cis_reports/970 Processing Data-intensive Workflows in the Cloud WPE-II Written Report